Design Considerations of a Flexible Data Stream Processing Middleware

Techniques for efficient and distributed processing of huge, unbound data streams have made some impact in the database community. Distributed data stream processing systems have emerged providing a distributed environment to process these potentially unbound streams of data by a set of processing nodes. A wide range of real-time applications process stream-based data. Sensors and data sources, such as position data of moving objects, continuously produce data that is consumed by, e.g., location-aware applications. Depending on the domain of interest, the processing of such data often depends on domain-specific functionality. For instance, an application which visualizes stream-based data has stringent timing constraints, or may even need a specific hardware environment to smoothly process the data. Furthermore, users may add additional constraints. E.g., for security reasons they may want to restrict the set of nodes that participates in processing. In this paper we review context-aware applications which, despite their different application fields, share common data processing principles. We analyse these applications and extract common requirements which data stream processing systems must meet to support these applications. Finally, we show how such applications are implemented using NexusDS, our extensible stream processing middleware.